Register the S3 request signer on the session - #3783
Conversation
S3FileSystem creates the client that issues requests lazily inside the running event loop, so a handler attached to the eagerly constructed fs.s3 is never inherited by it. With signature_version set to UNSIGNED, requests then reach S3 with no Authorization header and are rejected. Registering on the session means every client it creates carries the signer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolves a conflict in _s3(): main added s3_additional_kwargs handling for server-side encryption while this branch replaced the post-construction signer registration with registration on an AioSession. Both apply — the additional kwargs are added to s3_fs_kwargs before the session is attached.
|
This also fixes #3896, which is a second, independent reason the Your change closes it because it drops the
Both are in They are on #3896 if useful. Happy to raise them against this PR instead so the fix and its regression cover land together — your call, and no action needed from you either way. |
|
Worth checking before this lands: it changes The obvious remedy is to memoise the session, and I'd hold off on that: So the caching cost is real but the fix for it is gated on making the signer thread-safe. Happy to raise that as its own issue with the measurements if it would help — it's independent of this PR, and this PR doesn't make it worse. |
`_s3()` unregisters the S3 request signer and re-registers it on an event emitter fsspec caches and every thread shares. A request signed in that window is sent with no Authorization header, because `signature_version` is already UNSIGNED, and the store answers 403. Two tests, no credentials and no network. Both fail on main and pass once the `unregister` is dropped: test_s3_leaves_a_signer_installed_while_reconfiguring_a_shared_client test_the_signer_stays_installed_while_another_thread_reconfigures_s3 The window is two adjacent statements, so sampling for it blind is a coin flip -- 20,000 observations from another thread caught it zero times. The second test holds it open by delaying only the re-registration: the unregister, the emitter and the observing thread are real, and dropping the unregister closes the window whatever the delay. Replaces tests/io/test_fsspec_signer_registration.py, which asserted the defective behaviour and so passed on main and failed once fixed. Placed beside the existing signer tests, as apache#3783 places its own. Verified against apache#3783, which removes the same `unregister` while registering the signer on an AioSession: both tests pass with it applied.
Closes #3625
Rationale for this change
FsspecFileIOremote request signing (s3.signer=S3V4RestSigner) silently stops signing oncurrent
aiobotocore/s3fs._s3()builds the filesystem first and then attaches the signer tofs.s3:Touching
fs.s3forces one eager client, but the client that actually issues requests is createdlazily inside the running event loop by
S3FileSystem.set_session, and it does not inherithandlers registered on that earlier client. Since
config_kwargs["signature_version"]is set toUNSIGNED, requests then go out with noAuthorizationheader and S3 rejects them withInvalidRequest: The authorization mechanism you have provided is not supported.This change registers the signer on an
aiobotocoresession and passes that session toS3FileSystem, so every client the session creates carries the handler.register_lastand theunique_idare kept, so ordering relative to the stockbefore-sign.s3handlers is unchanged.The
unregistercall is dropped because a freshly created session has nothing registered underthat id.
Credit for the diagnosis and the proposed approach goes to the issue reporter.
Are these changes tested?
Yes.
test_s3v4_rest_signer_registered_on_sessionis added totests/io/test_fsspec.py. Itbuilds the
FileIOwith a signer configured, takes the session handed toS3FileSystem, emitsbefore-sign.s3on it, and asserts the REST signer ran — the request URL is rewritten and anAuthorizationheader is set. Onmainit fails withKeyError: 'session', since no session ispassed today.
The mechanism was also confirmed directly against
s3fs2026.4.0 andaiobotocore3.8.0 bydriving
set_session()and inspecting the handlers on the client that ends up issuing requests:make lint— passes, including mypymake test— 3804 passed, 3 skippedNot verified end to end against a live REST catalog with remote signing and real S3, as that
needs credentials and an S3 endpoint that enforces SigV4;
motodoes not. The end-to-endconfirmation in the issue report covers that path.
Are there any user-facing changes?
No API change. Remote signing starts working again on current dependency versions, so
FsspecFileIOusers on a modernaiobotocorewill see requests signed rather than rejected.AI assistance
Claude Code was used to investigate the issue, write the change and the test, and run the
verification described above. The full diff was reviewed before submission.